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(54) IMAGE PROCESSOR 

(57)Abstract: 

PROBLEM TO BE SOLVED: To exactly overlap images 
by correcting the images so as to match the object 
positions of the respective images based on the feature 
position information of input images and the deviation 
between the respective coordinates of position 
information at correspondent points. 
SOLUTION: A feature point setting part 33 sets the 
plural feature points (coordinates) of the input image. 
Image data around the feature points are segmented as 
a template and inputted to a correspondent point search 
part 35. The correspondent point search part 35 
searches a point corresponding to the template in an 
image different from the image inputted to the feature 
point setting part 33 and outputs the correspondent point 

coordinate position to a correspondent point position memory 36. A correction parameter 
calculating part 37 reads out the feature point coordinate in a memory 34 and the 
correspondent point coordinate in a memory 36 and calculates the position relation (parallel 
moving amount or rotating angle) between two images. An interpolation operating part 38 
corrects the images based on the inputted position relation and outputs them to an image 
synthesizing part 39. The image synthesizing part 39 executes dynamic range expanding 
processing or object field depth expanding processing. 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1 .This document has been translated by computer. So the translation may not reflect the 
original precisely. 

2.**** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



CLAIMS 

[Claim(s)] 

[Claim 1]An image processing device which performs alignment processing between two or 
more pictures which photoed almost same composition by a different photographing condition, 
comprising: 

A seek-area setting-out means to divide at least one picture of an inputted picture into two or 
more region divisions, and to provide a seek area showing the feature of a picture of the focus 
in said each region division. 

The 1st position information extraction means that relates characteristic position information on 
a photographic subject within said composition to coordinates over each region division of said 
division picture, and extracts it from a seek area of each of said focus based on characteristic 
quantity of said picture. 

The 2nd position information extraction means extracted as position information which related 
a point acquired by searching for a point corresponding to said characteristic position 
information to each of said inputted images other than said divided picture of corresponding to 
coordinates over the picture. 

A corrected parameter detection means to detect a corrected parameter based on a gap 
between each coordinates of said extracted characteristic position information and position 
information on said point of corresponding, and a picture compensation means which amends 
a picture based on a detected corrected parameter so that an object position of each picture 
may be in agreement. 

[Claim 2]The image processing device according to claim 1, wherein position information 
extracted by the said 1st and 2nd position information extraction means is the position 
information which excepted position information relevant to coordinates of a region division 
where an object which moves in inside of said picture was detected. 
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[Claim 3]Position information extracted by the said 1st and 2nd position information extraction 
means, a pixel with which a high frequency component when orthogonal transformation was 
carried out was comparatively saturated to each region division within said picture -- 
comparatively on the basis of a value of a difference of average value of a pixel value, and the 
greatest and minimum pixel value, and either of the contrast. The image processing device 
according to claim 2 extracting coordinates related position information over a desired region 
division from said inputted picture. 



[Translation done.] 
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DETAILED DESCRIPTION 

[Detailed Description of the Invention] 
[0001] 

[Field of the lnvention]This invention relates to the image processing device which performs 
alignment amendment so that a photographic subject may be in agreement, when the device 
which combines the picture beyond the dynamic range and depth of field of the input device is 
started from two or more pictures which photoed the same photographic subject by a different 
photographing condition, especially each picture is piled up. 
[0002] 

[Description of the Prior Art]By combining two or more pictures which generally photoed the 
same photographic subject by different exposure. The image processing technique which 
creates the picture which has a dynamic range of the range more than the peculiar dynamic 
range of image sensors, such as CCD (Charge Coupled Device). (For example, JP,63- 
232591, A) Two or more pictures which photoed the same photographic subject in a different 
focal position are combined, and there is an image processing technique (JP,1-309478,A) 
which creates the picture which has depth of field deeper than the depth of field of an input 
optical system. 

[0003]Art which creates this wide dynamic range image is realized with the composition shown 
in drawing 12 (a). Picture illustrated. A and B are the pictures which changed and photoed 
exposure, respectively. 

The relation between the incident light quantity which enters into an image sensor, and the 
signal value of the picture acquired with image input apparatus becomes like "[A] long time 
exposure" of drawing 12 (b), and "[B] short-time exposure." 

[0004]ln the adding machine 1 1 , the pixel value of the pictures A and B is added and the value 
shown in drawing 12 (b) as a T[ [C] summed signal' 7 is memorized by the frame memory 12. The 
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shape of this "[C] Summed signal" is determined by the exposure ratio of the picture of two 
sheets. In the linear transform part 13, the incident light quantity I is presumed from the 
summed signal value S, and it changes into the luminance-signal value Y from the signal value 
of R, B, and G each color in the matrix circuit 15. 

[0005]And in the luminosity compression zone 16, a luminance value is compressed and 
compressed luminance-signal value Y' is outputted so that presumed incident light quantity 
may be settled in a fixed gradation number. R, G, and B which Y7Y is calculated in the divider 
17 and are the outputs of the linear transform part 16 - it is multiplied with each signal and 
multiplier 18, and a result is stored in the frame memory 19. 

[0006]Since this image processing technique uses two or more image data, it can acquire the 
picture which has good color reproduction, and few noises compared with the case where tone 
curve processing of the short-time exposure picture is only carried out. 
[0007]The image processing technique which creates the picture which has depth of field 
deeper than the depth of field of the input optical system mentioned above is realized with the 
composition shown, for example in drawing 13 . First, weighting addition is carried out with the 
adding machine 21, and the picture which changed the focal position and was photoed is 
memorized by the frame memory 22. If weighting addition of the picture which changed the 
focal position is carried out according to a focal position, it is known that the Japanese quince 
of the whole picture will be equalized. 

[0008]So, in the recovery filter setting circuit 25, the recovery filter of the Japanese quince in 
the picture after addition is determined, and it outputs to the matrix arithmetic circuit 23. Here, 
since it is not a portion in connection with the essence of this invention, the preparation method 
of a Japanese quince recovery filter is omitted. In the matrix arithmetic circuit 23, a recovery 
filter is made to act on an addition picture, and a result is stored in the frame memory 24. As a 
result, the picture stored in the frame memory 24 is a picture whose depth of field is deeper 
than an imaging optical system. 
[0009] 

[Problem(s) to be Solved by the lnvention]Once the image processing technique mentioned 
above adds or adds [ weighting ] the photoed picture for every pixel, it performs dynamic- 
range-compression processing and a series of processings of recovery. Therefore, when the 
picture which is the target of addition had caused the position gap mutually, it shifted, when it 
added and an exact result was not obtained. 

[0010]Therefore, in the case of dynamic range expanding processing, . Since nondestructive 
readout is possible, use the apparatus using the CMD (Charge Modurated Device) element 
which does not cause a position gap while photoing a picture by two or more exposure 
theoretically. In depth-of-field expanding processing, as it said that a firm stage was required, 
the expensive business-use device was needed. When changing and photoing a focal position, 
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change of the delicate magnification for every focal position also influences. 
[001 1]With a digital camera, a scanner, and a tripod which a general user uses, what [ a thing ] 
that cannot expect accuracy simple occupies most to it compared with commercial equipment. 
Therefore, the advantage of the image processing technique mentioned above was not fully 
able to be employed efficiently. 

[0012]For example, when a camera moves when carrying and photoing a digital camera to a 
tripod, and pushing a shutter, or capturing the image photoed with the film-based camera by a 
film scanner, it is shifting mutually by the method of insertion of a film etc. 
[0013]Then, an object of this invention is to provide the image processing device amended 
with the value of the parameter which detects the physical relationship between several 
pictures from which photographing conditions, such as exposure and a focal position, differ, 
and shows the detected physical relationship so that a picture may lap correctly. 
[0014] 

[Means for Solving the Problem]An image processing device whose this invention is 
characterized by that an image processing device comprises the following and which performs 
alignment processing between two or more pictures which photoed to achieve the above 
objects almost same composition (photographing area) by a different photographing condition. 
A seek-area setting-out means to divide at least one picture of an inputted picture into two or 
more region divisions, and to provide a seek area showing the feature of a picture of the focus 
in said each region division. 

The 1st position information extraction means that relates characteristic position information on 
a photographic subject within said composition to coordinates over each region division of said 
division picture, and extracts it from a seek area of each of said focus based on characteristic 
quantity of said picture. 

The 2nd position information extraction means extracted as position information which related 
a point acquired by searching for a point corresponding to said characteristic position 
information to each of said inputted images other than said divided picture of corresponding to 
coordinates over the picture. 

A corrected parameter detection means to detect a corrected parameter based on a gap 
between each coordinates of said extracted characteristic position information and position 
information on said point of corresponding, and a picture compensation means which amends 
a picture based on a detected corrected parameter so that an object position of each picture 
may be in agreement. 

[0015]One in two or more inputted pictures is chosen, this picture is divided into two or more 
fields, and an image processing device of the above composition sets up the focus from 
characteristic quantity of that region division, and memorizes a coordinates position of that 
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focus. As opposed to each another inputted image which was not chosen based on 
coordinates of the focus, searching for a point of corresponding and extracting a coordinates 
position of the corresponding points -- physical relationship (the amount of parallel translation.) 
between [ coordinates and corresponding-points coordinates of the focus to ] pictures Angle of 
rotation is computed, and after performing alignment amendment so that a picture of one side 
or both may be amended and a photographic subject may agree, dynamic-range-compression 
processing, depth-of-field expanding processing, etc. are performed. 
[0016] 

[Embodiment of the lnvention]Hereafter, with reference to drawings, the embodiment of this 
invention is described in detail. The example of composition of the image processing device as 
a 1st embodiment by this invention is shown and explained to drawing 1. 
[0017]ln this embodiment, the picture which is photoed with the picture and film-based camera 
which were photoed with the digital camera which is not illustrated, and was photographed with 
the scanner etc. is recorded on storages, such as a hard disk, a portable floppy disk, and a 
memory card. 

[0018]The image data regenerating section 31 which this embodiment reads the picture 
currently recorded on said hard disk, the floppy disk, the memory card, etc. if it divides roughly, 
and processes extension etc., The picture amendment part 30 which performs interpolating 
calculation and amends the physical relationship of the photographic subject of a picture by the 
method of mentioning later from the image data regenerating section 31 to an inputted image, 
It comprises the image synthesis section 39 which performs dynamic range expansion 
mentioned above and depth-of-field expansion to the picture by which physical relationship 
was amended by said picture amendment part 30. 

[0019]The frame memory 32 said picture amendment part 30 remembers temporarily the 
image data to which elongation processing was performed by the image data regenerating 
section 31 to be, The focus set part 33 which sets up two or more focus (coordinates) later 
mentioned to one selected picture, The corresponding-point-searching part 35 which searches 
for the point of corresponding in another picture from the image data of the focus 
circumference started as the focus position memory 34 which memorizes the focus 
coordinates, and focus coordinates and a template, The corresponding-points position memory 
36 which memorizes the acquired corresponding-points coordinates position, and the 
corrected parameter calculation part 37 which computes the physical relationship (the amount 
of parallel translation, angle of rotation) between the pictures of two sheets from focus 
coordinates and corresponding-points coordinates, It comprises the interpolating calculation 
part 38 which amends the picture of one side or both and is outputted to the image synthesis 
section 39 from the inputted physical relationship. 
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TECHNICAL FIELD 

[Field of the lnvention]This invention relates to the image processing device which performs 
alignment amendment so that a photographic subject may be in agreement, when the device 
which combines the picture beyond the dynamic range and depth of field of the input device is 
started from two or more pictures which photoed the same photographic subject by a different 
photographing condition, especially each picture is piled up. 
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PRIOR ART 

[Description of the Prior Art]By combining two or more pictures which generally photoed the 
same photographic subject by different exposure. The image processing technique which 
creates the picture which has a dynamic range of the range more than the peculiar dynamic 
range of image sensors, such as CCD (Charge Coupled Device). (For example, JP,63- 
232591 ,A) Two or more pictures which photoed the same photographic subject in a different 
focal position are combined, and there is an image processing technique (JP,1-309478,A) 
which creates the picture which has depth of field deeper than the depth of field of an input 
optical system. 

[0003]Art which creates this wide dynamic range image is realized with the composition shown 
i n drawing 12 (a). Picture illustrated. A and B are the pictures which changed and photoed 
exposure, respectively. 

The relation between the incident light quantity which enters into an image sensor, and the 
signal value of the picture acquired with image input apparatus becomes like "[A] long time 
exposure" of drawin g 12 (b), and "[B] short-time exposure." 

[0004]ln the adding machine 1 1 , the pixel value of the pictures A and B is added and the value 
shown in drawing 12 (b) as a "[C] summed signal" is memorized by the frame memory 12. The 
shape of this "[C] Summed signal" is determined by the exposure ratio of the picture of two 
sheets. In the linear transform part 13, the incident light quantity I is presumed from the 
summed signal value S, and it changes into the luminance-signal value Y from the signal value 
of R, B, and G each color in the matrix circuit 15. 

[0005]And in the luminosity compression zone 16, a luminance value is compressed and 
compressed luminance-signal value Y' is outputted so that presumed incident light quantity 
may be settled in a fixed gradation number. R, G, and B which Y7Y is calculated in the divider 
17 and are the outputs of the linear transform part 16 - it is multiplied with each signal and 
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multiplier 18, and a result is stored in the frame memory 19. 

[0006]Since this image processing technique uses two or more image data, it can acquire the 
picture which has good color reproduction, and few noises compared with the case where tone 
curve processing of the short-time exposure picture is only carried out. 
[0007]The image processing technique which creates the picture which has depth of field 
deeper than the depth of field of the input optical system mentioned above is realized with the 
composition shown, for example in drawing 13 . First, weighting addition is carried out with the 
adding machine 21, and the picture which changed the focal position and was photoed is 
memorized by the frame memory 22. If weighting addition of the picture which changed the 
focal position is carried out according to a focal position, it is known that the Japanese quince 
of the whole picture will be equalized. 

[0008]So, in the recovery filter setting circuit 25, the recovery filter of the Japanese quince in 
the picture after addition is determined, and it outputs to the matrix arithmetic circuit 23. Here, 
since it is not a portion in connection with the essence of this invention, the preparation method 
of a Japanese quince recovery filter is omitted. In the matrix arithmetic circuit 23, a recovery 
filter is made to act on an addition picture, and a result is stored in the frame memory 24. As a 
result, the picture stored in the frame memory 24 is a picture whose depth of field is deeper 
than an imaging optical system. 
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EFFECT OF THE INVENTION 

[Effect of the lnvention]As explained in full detail above, according to this invention, the image 
processing device amended with the value of the parameter which detects the physical 
relationship between several pictures from which photographing conditions, such as exposure 
and a focal position, differ, and shows the detected physical relationship so that a picture may 
lap correctly can be provided. 
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TECHNICAL PROBLEM 

[Problem(s) to be Solved by the lnvention]Once the image processing technique mentioned 
above adds or adds [ weighting ] the photoed picture for every pixel, it performs dynamic- 
range-compression processing and a series of processings of recovery. Therefore, when the 
picture which is the target of addition had caused the position gap mutually, it shifted, when it 
added and an exact result was not obtained. 

[0010]Therefore, in the case of dynamic range expanding processing, . Since nondestructive 
readout is possible, use the apparatus using the CMD (Charge Modurated Device) element 
which does not cause a position gap while photoing a picture by two or more exposure 
theoretically. In depth-of-field expanding processing, as it said that a firm stage was required, 
the expensive business-use device was needed. When changing and photoing a focal position, 
change of the delicate magnification for every focal position also influences. 
[001 1]With a digital camera, a scanner, and a tripod which a general user uses, what [ a thing ] 
that cannot expect accuracy simple occupies most to it compared with commercial equipment. 
Therefore, the advantage of the image processing technique mentioned above was not fully 
able to be employed efficiently. 

[0012]For example, when a camera moves when carrying and photoing a digital camera to a 
tripod, and pushing a shutter, or capturing the image photoed with the film-based camera by a 
film scanner, it is shifting mutually by the method of insertion of a film etc. 
[0013]Then, an object of this invention is to provide the image processing device amended 
with the value of the parameter which detects the physical relationship between several 
pictures from which photographing conditions, such as exposure and a focal position, differ, 
and shows the detected physical relationship so that a picture may lap correctly. 
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DESCRIPTION OF DRAWINGS 

[Brief Description of the Drawings] 

[Drawing 1]lt is a figure showing the example of composition of the image processing device 
as a 1st embodiment. 

[D rawing 2] It is a figure showing the example of composition of the focus set part shown in 

drawing 1 . 

[Drawing 3]The figure and drawing 3 (b) which drawing 3 (a) shows the region division where 
plurality divided the picture, and the focus seek area set as each region division are a figure for 
explaining the focus. 

[Drawing 4] lt is a figure showing the example of composition of the corresponding-point- 
searching part shown in dra wing 1 . 

[Drawing 5] The figure in which drawing 5 (a) shows the example of composition of the image 
processing device as a 2nd embodiment, and drawing 5 (b) are the figures showing the 
example of composition of a territorial extension part. 

[Drawing SjDrawing 8 (a) is a figure in which the figure showing an example of the appointed 
field displayed on the picture and drawing 6 (b) show the focused position displayed on the 
picture. 

[Drawing 7]lt is a figure showing the example of composition of the image processing device 
as a 3rd embodiment. 

[Drawing 8] It is a figure showing the example of composition of the territorial extension part 
used for the image processing device as a 4th embodiment. 

[Drawing 9]Drawing 9 (a) is a figure for a figure for the figure and drawing 9 (b) in which the 
inputted image divided into two or more region divisions is shown to explain specification of the 
focus set area by scan, and drawing 9 (c) to explain specification of the focus set area by 
extension. 

[Drawing 1 0] 1 1 is a figure for explaining specifying the rectangle extended from the designated 



http://www4.ipdl. inpit.go.ip/cgi-bin/tran_web_cgi_eiie?atw_u=http%3A%2F%2Fwww4.ip... 3/11/2009 



JP,11-185018,A [DESCRIPTION OF DRAWINGS] 



Page 2 of 2 



point as a focus set area. 

[Drawing 11] lt is a figure showing the example of composition of the image processing device 
as a 5th embodiment. 

[Drawing 12]lt is a figure showing the example of composition for creating the conventional 
wide dynamic range image. 

[Drawing 1 3] It is a figure showing the example of composition for creating the picture which 
has the conventional deep depth of field. 
[Description of Notations] 

30 -- Picture amendment part 

31 -- Image data regenerating section 

32 - Frame memory 

33 -- Focus set part 

34 -- Focus position memory 

35 -- Corresponding-point-searching part 

36 -- Corresponding-points position memory 

37 - Corrected parameter calculation part 

38 -- Interpolating calculation part 

39 -- Image synthesis section 
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MEANS 

[Means for Solving the Problem]An image processing device whose this invention is 
characterized by that an image processing device comprises the following and which performs 
alignment processing between two or more pictures which photoed to achieve the above 
objects almost same composition (photographing area) by a different photographing condition. 
A seek-area setting-out means to divide at least one picture of an inputted picture into two or 
more region divisions, and to provide a seek area showing the feature of a picture of the focus 
in said each region division. 

The 1st position information extraction means that relates characteristic position information on 
a photographic subject within said composition to coordinates over each region division of said 
division picture, and extracts it from a seek area of each of said focus based on characteristic 
quantity of said picture. 

The 2nd position information extraction means extracted as position information which related 
a point acquired by searching for a point corresponding to said characteristic position 
information to each of said inputted images other than said divided picture of corresponding to 
coordinates over the picture. 

A corrected parameter detection means to detect a corrected parameter based on a gap 
between each coordinates of said extracted characteristic position information and position 
information on said point of corresponding, and a picture compensation means which amends 
a picture based on a detected corrected parameter so that an object position of each picture 
may be in agreement. 

[0015]One in two or more inputted pictures is chosen, this picture is divided into two or more 
fields, and an image processing device of the above composition sets up the focus from 
characteristic quantity of that region division, and memorizes a coordinates position of that 
focus. As opposed to each another inputted image which was not chosen based on 
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coordinates of the focus, searching for a point of corresponding and extracting a coordinates 
position of the corresponding points -- physical relationship (the amount of parallel translation.) 
between [ coordinates and corresponding-points coordinates of the focus to ] pictures Angle of 
rotation is computed, and after performing alignment amendment so that a picture of one side 
or both may be amended and a photographic subject may agree, dynamic-range-compression 
processing, depth-of-field expanding processing, etc. are performed. 
[0016] 

[Embodiment of the lnvention]Hereafter, with reference to drawings, the embodiment of this 
invention is described in detail. The example of composition of the image processing device as 
a 1st embodiment by this invention is shown and explained to drawing 1. 
[0017]ln this embodiment, the picture which is photoed with the picture and film-based camera 
which were photoed with the digital camera which is not illustrated, and was photographed with 
the scanner etc. is recorded on storages, such as a hard disk, a portable floppy disk, and a 
memory card. 

[0018]The image data regenerating section 31 which this embodiment reads the picture 
currently recorded on said hard disk, the floppy disk, the memory card, etc. if it divides roughly, 
and processes extension etc., The picture amendment part 30 which performs interpolating 
calculation and amends the physical relationship of the photographic subject of a picture by the 
method of mentioning later from the image data regenerating section 31 to an inputted image, 
It comprises the image synthesis section 39 which performs dynamic range expansion 
mentioned above and depth-of-field expansion to the picture by which physical relationship 
was amended by said picture amendment part 30. 

[0019]The frame memory 32 said picture amendment part 30 remembers temporarily the 
image data to which elongation processing was performed by the image data regenerating 
section 31 to be, The focus set part 33 which sets up two or more focus (coordinates) later 
mentioned to one selected picture, The corresponding-point-searching part 35 which searches 
for the point of corresponding in another picture from the image data of the focus 
circumference started as the focus position memory 34 which memorizes the focus 
coordinates, and focus coordinates and a template, The corresponding-points position memory 
36 which memorizes the acquired corresponding-points coordinates position, and the 
corrected parameter calculation part 37 which computes the physical relationship (the amount 
of parallel translation, angle of rotation) between the pictures of two sheets from focus 
coordinates and corresponding-points coordinates, It comprises the interpolating calculation 
part 38 which amends the picture of one side or both and is outputted to the image synthesis 
section 39 from the inputted physical relationship. 

[0020]ln the image processing method of this embodiment constituted in this way, processing 
of extension etc. is performed by the image data regenerating section 31, and the picture read 
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from said hard disk, the memory card, etc. is memorized by the frame memory 32 of the 
picture amendment part 30. Here, the data outputted from the image data regenerating section 
31 shall contain not only the pixel value of a picture but the additional data of image size, a 
color number, a color map, etc. 

[0021]Next, one in an inputted image is chosen (here, it is considered as the picture A), and it 
is inputted into the focus set part 33. In this focus set part 33, two or more focus (coordinates) 
is set up by the method of mentioning later, and focus coordinates are inputted into the focus 
position memory 34. The set-up focus coordinates are inputted also into the corresponding- 
point-searching part 35, and simultaneously, the image data of the focus circumference is 
started as a template, and is inputted into the corresponding-point-searching part 35. 
[0022]lt is among a picture other than the picture inputted into the focus set part 33 (here the 
picture B), and the corresponding-point-searching part 35 is searched for the point 
corresponding to the inputted template. The method of searching for corresponding points is 
mentioned later. 

[0023]As a result, a corresponding-points coordinates position is outputted to the 
corresponding-points position memory 36. In the corrected parameter calculation part 37, the 
focus coordinates of the focus position memory 34 and the corresponding-points coordinates 
of the corresponding-points position memory 36 are read, and the physical relationship (the 
amount of parallel translation, angle of rotation) between the pictures of two sheets is 
computed. And in the interpolating calculation part 38, from the inputted physical relationship, 
the picture of one side or both is amended and it outputs to the image synthesis section 39. 
[0024]Although the dynamic range expanding processing and depth-of-field expanding 
processing which were mentioned above are performed in this image synthesis section 39, 
since the target picture is a picture to which position ****** was carried out in the picture 
amendment part 30 of the preceding paragraph, the picture naturally piled up correctly is 
acquired as a resulting image. 

[0025]The composition of said focus set part 33 is shown and explained to drawing 2 . At this 
focus set part 33, although two or more feature point positions are determined, when many 
focus concentrates on a part of picture, the error of position ****** may become large in the 
other portion. So, in the region dividing part 41, in order to make it the focus suitably scattered 
in a picture, a picture is divided into two or more fields, and a focus seek area is further set as 
the inside. Here, the example which divided the whole uniformly is shown in drawing 3 (a). 
[0026]The small region which serves as a candidate of a template all over this field is moved, 
and the position from which characteristic quantity serves as the maximum is outputted as one 
of the focus. However, since processing time will start dramatically if it searches for the whole, 
it is desirable to set up a focus seek area smaller than the field divided like drawing 3 (a). The 
size of this focus seek area is beforehand decided on the basis of image size, for example like 
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5% of image size. Or it does not matter on the basis of the size of the divided small region. 
[0027]Next, in the template spotting part 42, the template candidate position in a focus seek 
area is determined, and the image data in a template candidate is started by the template 
extraction part 43 based on the position. At this time, a template candidate's position is moved 
in order within the feature seek area in the template spotting part 42. 

[0028]Since a template candidate's characteristic quantity change near in position may not be 
so large, in such a case, it is not that of ****** the whole pixel, it can also shift a template 
candidate's position at intervals of several pixels, and can reduce computational complexity as 
a result. 

[0029]ln the characteristic quantity calculation part 44, a template candidate's characteristic 
quantity is calculated and the template position from which characteristic quantity serves as 
the maximum by the characteristic quantity judgment part 45 is determined. In the template 
extraction part 46, the data of the position circumference in which characteristic quantity 
became the maximum is extracted as a template. As for this template extraction part 46, the 
template extraction part 43 may serve as that role. As this characteristic quantity, pixel value 
distribution as shown, for example in (1) type can be used. 
[0030] 
[Equation 1] 

T = £ |ai - A | (1) 

Here, ai is a pixel value, A is the pixel value average in a template candidate, and peace is 
taken to the whole inside of a template candidate. Although a concrete formula does not 
mention, they are the secondary pixel value distribution and a value also with an edge 
detection result preferred as characteristic quantity. At this time, said value T is a following 
formula, [0031] 
[Equation 2] 

T < T th (2) 

** - like, when smaller than a certain default value Tth, it does not adopt noting that it is 
unreliable. For example, although the accuracy of matching which change of a pixel value 
generally mentions later in a small portion falls like the portion of the empty in drawing 3 (b), by 
the method of this invention, the focus is not set to a focus candidate position with the value T 
like (2) types, but the accuracy reduction of a physical relationship parameter can be 
prevented. 

[0032]Drawing 4 is a figure explaining the composition of the corresponding-point-searching 
part 35. In the search area set part 51, the position and size of a field which search for 
corresponding points are set up on the basis of the focus coordinates read from the focus set 
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part 33. In the correlation operation part 52, based on the information in the search area set up 
by the search area set part 51 , The image data of the circumference of a corresponding-points 
candidate position is started from the image data in the frame memory 32b so that it may have 
the same size as the template inputted from the focus set part 33, and a correlation value with 
the template data inputted from the focus set part 33 is calculated. 
[0033]ln the correlation value judgment part 53, a template and a position with the highest 
correlation which were extracted by the template extraction part 46 are judged, and it outputs 
to the corresponding-points position memory 36 by making a result into a corresponding-points 
position. 

[0034]Although a corresponding-points candidate generally moves for every pixel at this time 
and a correlation value is calculated, in order to detect a corresponding-points position in 
accuracy below a pixel, with reference to the surrounding correlation value, interpolation may 
determine a corresponding-points position. As the correlation value, an absolute value sum of 
difference shown in the following (3) types, cross correlation shown in (4) types, normalization 
cross correlation shown in (5) types, etc. can be used. 
[0035] 
[Equation 3] 

E = S|ai-bi| (3) 
i 

E = £ ( ai - A ) ■ (bi - B ) (4) 
i 

-A ) ( bi -B ) 

E = -i (5) 

ct a ■ a B 

Here, bi and B are the pixel values and averages in the small region where it moves in the 
search area of the picture B, sigma A and sigma B are the standard deviation of template data, 

2 2 

and sigma A and sigma B serve as the variance. 

[0036]lf it calculates only about a certain representative color when calculating the 
aforementioned (2) - (4) type, it can process at high speed. If the correlation value of R, G, and 
B each color is added and it searches for corresponding points, it will become possible to 
perform corresponding-points search to high degree of accuracy more. 
[0037]And in the corrected parameter calculation part 37, physical relationship between the 
pictures A and B is determined from focus coordinate data and corresponding-points 
coordinate data. Parameters for which it asks are costheta of (6) types, sintheta, sx, sy, and M. 

[0038] 
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[Equation 4] 

(tC\ (cos 9 - sin Q\ (M ■ x> fs x "> 

UJ = Un9 coseJ-U.yJHsyJ W 

Here, a feature point position, and (X, Y) show a corresponding-points position, and M is [ (x 
y) ] equivalent to the relative magnification between pictures. Since two or more data is stored 
in said focus position memory 34 and the corresponding-points position memory 36, if it asks 
for a parameter in least square, it will become like (7) types. 
[0039] 
[Equation 5] 

A c = M • cos 9 = (xX + yY - XX - yy) /(tj 2 - a y 2 ) 
A a = M ■ sin 6 = (xY - yX + yX - xy) /(<j x 2 - o y 2 ) 
Sjt - X - M - (x cos fl - y sin G) 
S y = Y — M • (x sin 6 + y cos 0) 

-l N _i N _i N i H 

i i i i 

— 1 N — 1 N 

XX = - • J) Xi • X ir yY = - - Y, Yi • Y if P> 
1 N i N 

«Y = - Exi Yi, yX = - Sxj- Yi 

It is here and N is the number of the focus. (8) From a formula, M, costheta, and sintheta are 
determined like (9) types. 
[0040] 
[Equation 6] 

A c 2 + A s 2 = M 2 (8) 
M = VA C 2 + A s 2 , cos 9 = ^£ , sin 9 = ^ (9) 

[0041]When it can be assumed that there is no magnification change in an object image since 
original costheta and sintheta have a relation of cos 2 theta+sin 2 theta= 1, by the result of (9) 
types. It is better to amend costheta and sintheta like (11) types, when it is referred to as M= 
1.0 and becomes a relation like (10) types, sx and sy are calculated from amended costheta 
and sintheta. 
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[0042] 
[Equation 7] 



[0043]Although (6) - (9) type mentioned above made the arbitrary starting points the center of 
rotation of the picture, you may think that setting out and the searched center of gravity of the 
focus and each corresponding-points position are in agreement. The centroid position (here 
(/x, /y), / is used as a sign meaning an average.) of the focus as it is in the image information of 
each formula, in each formula, this average is expressed as an exaggerated bar - the centroid 
position of corresponding points -- (/X and /Y) - it is come out and given -- costheta, sintheta, 
sx, and sy are given by (12) and (13) formulas, respectively. 
[0044] 
[Equation 8] 

cos 8 = fix + yy)/^x 2 + y 2 j 



•in 8 = (xY - yx) /(x 2 + y 2 ) 



S y = Y - y (13) 

[0045]As explained above, a general user can also do the maximum use of the effect of 
dynamic range expanding processing and depth-of-field expanding processing, without using 
expensive business-use equipment, in order for this embodiment to extract two or more focus 
out of a picture and to perform position ****** automatically. However, when the distortion 
aberration has arisen in the object image of processing, it is necessary to amend beforehand 
by arbitrary techniques. 

[0046]Next, a 2nd embodiment by this invention is described. This embodiment differs in the 
composition inside the focus set part 33 as compared with a 1st embodiment mentioned 
above. 

[0047] Drawing 5 (a) is a figure showing the notional composition of the focus set part 33. This 
composition differs in that the territorial extension part 81 is added compared with the 
composition shown in drawing 1 Drawing 5 (b) is a figure showing the example of composition 
of the territorial extension part 81 . In this territorial extension part 81 , two or more fields to 
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observe are directed in the field directions part 83 to the inputted image data. The focus is set 
as the inside of the field specified here by the above-mentioned method. 
[0048]ln this display control part 84, a specified field (data from the region memory 85 
mentioned later) is overlaid on inputted image data, for example, as shown in drawing 6 (a) 
and (b), and that data is displayed on the indicator 82. A display of attachment in PC may be 
sufficient as this indicator 82, and it may provide a thing for exclusive use independently. You 
may point to the field directions part 83 with a mouse of attachment in PC, and it may be 
provided independently. Although said display control part 84 was explained as a part of 
composition of the territorial extension part 81, composition that a function of operating 
systems (OS), such as a personal computer, is used may be sufficient as it. 
[0049]ln the field directions part 83, an observation region may be specified with coordinates in 
a picture. And the region memory part 85 memorizes a position and size of a field directed in 
the field directions part 83, and outputs it to the region dividing part 41 . In the region dividing 
part 41, the focus is set up based on region information from the region memory part 85 by a 
method described above inside a directed field. 

[0050]Since raise alignment accuracy of a portion which a user wants to observe, and things 
are made by composition of this embodiment and focus search is not needed in a portion 
which is not so required, computation time can be shortened. 

[0051]When a photographic subject which moves is included in a taken image, if human being, 
a vehicle, etc. use a picture of a portion which moved for calculation, an error will produce 
them in alignment of the whole picture, but it can avoid using data of such a portion in 
composition of this example gestalt. 

[0052]lt is possible to point to a specific field among two or more fields to which it pointed in 
the field directions part 83, and only for the inside to improve alignment accuracy of the 
specific region further by setting up many focus from other fields. 

[0053]in the case of dynamic range expansion art, compared with a case where a picture is 
chosen and performed at random, accuracy can be improved by performing the above- 
mentioned operation in order of exposure value between pictures which align in the case of 
depth-of-field expansion art, and adjoin it in a picture in order of a focal position. 
[0054] Drawing 7 is a figure showing a 3rd embodiment of this invention. Composition of the 
picture amendment part 30 shown in drawing 7 is added to composition of the picture 
amendment part 30 shown in drawing 1, and the indicator 82, the display control part 84, and 
the focus directions part 90 are contained. Two or more feature point positions are set up by 
an above-mentioned method, and are memorized by the focus memory 34. 
[0055]Focus coordinates memorized by this focus memory 34 are sent to the focus directions 
part 90, and that position is overlaid by the indicator 82 on an object image through the display 
control part 84 (for example, drawing 3 (b)). 
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If the focus is set up on a photographic subject (movable matter object) in which human being, 
a vehicle, etc. move at this time, it will become a cause of an error of a correction factor 
calculated by (6) - (10) formula. So, in the focus directions part 90, when a movable matter 
object is included, the object is detected, and the focus set up on a movable matter object is 
excepted from data used for calculation, and deletes data from the focus memory 34. 
[0056]A user judges displayed data and this operation may be made to ** data excepted from 
the focus directions part 90. Data of the focus memory 34 and the corresponding-points 
memory 36 is simultaneously displayed on the indicator 82 after corresponding point 
searching, and when a user judges that correspondence cannot be taken, you may make to 
except the data into business which can be directed by the focus directions part 90. 
[0057]Next, a 4th embodiment by this invention is described with reference to drawing i 8. This 
4th embodiment is changed into composition as shows drawing 8 the territorial extension part 
81 of drawing 5 . 

[0058]The territorial extension part 81 shown in drawing 8 comprises the field directions part 
83, the region memory 85, and the state judging part 101. According to this embodiment, a 
range specified as a focus set area based on statistical and physical character of a picture is 
determined. 

[0059]First, the whole picture is divided into a size which was able to be beforehand decided 
like drawing 9 (a). Let size divided at this time be a larger range than a template started by a 
focus set part. 

[0060]lt is judged whether each divided portion is sent to the state judging part 101, and is 
specified as a focus set area. In the state judging part 101, when inputted image data is 
scanned and a saturated pixel is over a fixed rate, it is judged that it is not suitable as a focus 
set area. 

[0061]ln the state judging part 101 , when it is over a threshold with average value of input 
data, you may judge that it is not suitable as a focus set area. The field directions part 83 
outputs image data of a divided field to the state judging part 101 in order like drawing 9 (b), 
and when it is judged that it is suitable as a focus set area, it records including the range on 
the region memory 85. 

[0062]ln explanation mentioned above, although judged for every region division, you may 
make it the whole range currently recorded on the region memory 85 as image data outputted 
to a state judging part. A region division with a user is specified and it may be made to contain 
to the focus appointed field in an order from a field of the circumference of it in the field 
directions part 83, as shown in drawing 9 (c). 

[0063]One with a user is specified in the field directions part 83, a rectangle extended to the 
surroundings of it is assumed, and it may be made to judge the image data by the state 
judging part 101, as shown in drawing 10. 
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[0064]Although a saturated pixel value was used for a decision criterion, in the state judging 
part 101, orthogonal transformation, such as discrete Fourier transform of an inputted picture 
and a discrete cosine transform, may be judged by power of a high frequency component 
when it gives a picture, and an ingredient of wavelet transform. 

[0065]ln the state judging part 101, contrast of inputted data and a difference of the pixel value 
maximum and the minimum may be used for judgment. Next, a 5th embodiment by this 
invention is described with reference to drawing i 1 

[0066]This embodiment is an example from which a picture of three or more sheets is a 
processing object. Here, the same reference mark is given to a composition part equivalent to 
drawing 1 mentioned above while illustrating, and the explanation is omitted to it. 
[0067]ln this embodiment, image data reproduced by the image data regenerating section 31 
is inputted into one of the frame memories 32a and 32b by the image switching section 1 1 1 
which carries out switch operation with directions of the picture selection instructing part 1 14. 
This picture selection instructing part 114 chooses processing time of a device, etc. at its own 
discretion so that the inputted same photographic subject can process two or more photoed 
pictures sequentially, and it controls the picture changeover section 1 1 1 to be inputted by turns 
to two frame memories. 

[0068]The focus and corresponding points are detected by a method mentioned above from a 
picture stored in said frame memories 32a and 32b, and physical relationship (the amount of 
parallel translation, angle of rotation, and relative magnification) between pictures is computed 
by the corrected parameter calculation part 37. A parameter computed by the corrected 
parameter calculation part 37 is memorized by the corrected parameter storage parts store 
1 13 with picture ID and image comparison ID. 

[0069]Said corrected parameter converter 112 reads data memorized by the corrected 
parameter storage parts store 113, and changes it into the amount of parallel translation and 
angle of rotation which saw each data from a picture used as a standard of amendment 
specified beforehand. Then, in the interpolating calculation part 38, the corrected parameter 
converter 112 amends a picture according to a parameter seen from a correction reference 
picture. 

[0070]Amendment in data conversion and the interpolating calculation part 38 in the corrected 
parameter converter 112 may be performed for every operation of the image switching section 
111, and after computing physical relationship relative about all object images, a picture may 
be anew read into the frame memory 32 here. 

[0071]Although it was considered as the two frame memories 32 and explained that an 
inputted image was changed by the signal switching part 1 1 1 by this embodiment, a number of 
frame memories which can store all object images may be prepared. 
[0072]Although an above embodiment was described, the following inventions are also 
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included in this specification. 

(1) A picture is divided into two or more region divisions in several pictures from which a 
photographing condition differs, A focus seek area the same as each region division or small is 
provided all over said region division, Based on a statistics value of a picture, characteristic 
position information on a photographic subject is extracted from each focus seek area, It 
searches for a point of corresponding in a different picture from a picture which extracted 
characteristic position information on a photographic subject among said two or more pictures, 
An image processing device which detects corrected parameters (the rotation, movement, 
magnification change, etc. between pictures), and amends a picture based on the rotation, 
movement, and magnification change between pictures as said corrected parameter from said 
characteristic position information and corresponding position information on a point. 
[0073]This invention corresponds to a 1st embodiment, thereby, when you pile up two or more 
pictures, a more exact picture should pile up -- it can carry out now. 

(2) In an image processing device given in the aforementioned (1) paragraph, amend after 
changing a parameter so that the rotation, movement, and magnification change seen from a 
specific reference image may be expressed when amending a picture based on said rotation, 
movement, and magnification change. 

[0074]This invention corresponds to a 5th embodiment. Thereby, when piling up two or more 
pictures, suitable arbitrary corrected parameters can be set up to each picture, and a more 
exact picture according to a photography person's intention can be piled up. 

(3) An image processing device given in 1 restricting a field to divide to a part of picture about 
said two or more region divisions. 

[0075]This invention corresponds to a 2nd embodiment. By this, when piling up two or more 
pictures, speed of image processing improves, and a suitable field can be set up to each 
picture and a more exact picture according to a photography person's intention can be piled 
up. 

(4) Don't use for detection of the rotation, movement, and magnification change between 
pictures position information set up on a photographic subject which a position is moving 
among two or more pictures among characteristic position information on a photographic 
subject based on a statistics value of said picture in an image processing device given in the 
aforementioned (1) paragraph or (3) paragraphs. 

[0076]This invention corresponds to a 3rd embodiment. Since a field where a movable matter 
object exists is eliminated by this when piling up two or more pictures, a more exact picture 
can be piled up. 

(5) A restricted space which restricts a field to divide to a part of picture in an image processing 
device given in the aforementioned (3) paragraph, It is characterized by a thing of a pixel 
whose high frequency component when orthogonal transformation was carried out was 
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saturated comparatively comparatively restricted on the basis of a value of a difference of 
average value of a pixel value, and the greatest and minimum pixel value, and either of the 
contrast. 

[0077]This invention corresponds to a 4th embodiment. By this, when piling up two or more 
pictures, speed of image processing improves, and a suitable field can be automatically set up 
to each picture, and a more exact picture according to a photography person's intention can be 
piled up. 

(6) Extend a restricted space focusing on a specific point in an image processing device of a 
statement in the aforementioned (5) paragraph. 

[0078]This invention corresponds to a 4th embodiment. Since a suitable field is set up to each 
picture focusing on a specific point by this when piling up two or more pictures, a more exact 
picture can be piled up. 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1 .This document has been translated by computer. So the translation may not reflect the 
original precisely. 

2.**** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 
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[Drawing 1] 

30 mmm iESB 




{Drawing 2] 




[Drawing 3] 



>://www4.ipdl.inpit.go.ip/cgi-bin/tran_web^ 3/11/2009 



JP,11-185018,A [DRAWINGS] 



Page 2 of 5 




[Drawing 4] 
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[Drawing 6] 




[Drawing 8] 
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[Drawing 12] 
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[Drawing 13] 




[Translation done.] 



http://www4.ipdl.inpit.go. ip/cgi-bin/tra^ 3/11/2009 



